Flow-Based Unconstrained Lip to Speech Generation

Unconstrained lip-to-speech aims to generate corresponding speeches based on silent facial videos with no restriction head pose or vocabulary. It is desirable intelligible and natural speech a fast speed in unconstrained settings. Currently, handle the more complicated scenarios, most existing methods adopt autoregressive architecture, which optimized MSE loss. Although these have achieved promising performance, they are prone bring issues including high inference latency mel-spectrogram over-smoothness. To tackle problems, we propose novel flow-based non-autoregressive model (GlowLTS) break constraints achieve faster inference. Concretely, decoder by maximizing likelihood of training data capable generation. Moreover, devise condition module improve intelligibility generated speech. We demonstrate superiority our proposed method through objective subjective evaluation Lip2Wav-Chemistry-Lectures Lip2Wav-Chess-Analysis datasets. Our demo video can be found at https://glowlts.github.io/.